<html>
<head>
<title>OSPFv2 Testsuite - Documentation</title>
<link rel="stylesheet" href="testsuite.css" type="text/css">
</head>
<body>
<br><hr>
<div align="center"> 
<a href="http://ssfnet.d-jacob.net">Home</a> | 
<a href="index.html">Up</a> | 
<a href="selection.html">Previous</a> | 
<a href="ospfv2test.html">Next</a>
</div>
<hr><br>
<h1>Design of the Test Suite</h1>

The IOL test suite can be implemented with regards to the concepts discussed before. For 
OSPF testing, an upper tester is not needed, since OSPF does not provide any service to an 
upper layer. There are multiple lower testers, depending of the test case. Each tester can be 
represented by a router in the DML configuration. These routers run different protocol sessions, 
dependent of their function during the test. Most testers run normal protocol sessions (e.g. IP, 
ICMP and OSPF) in order to model the behavior of a normal router. This is sufficient to test 
most parts of the OSPF specification. Nevertheless, some tests require a non-standard behavior 
to verify, that a router correctly reacts on invalid or erroneous packets. These behaviours 
must be modeled in specialized testers, which must extend or modify the behavior of the standard 
protocol sessions.<p>

For the OSPF test suite, three classes of non-standard behavior are needed for testers. 
In general, all topologies are configured statically in SSFNet, which is sufficient for 
current applications. In order to test the behavior of OSPF when the topology changes 
(i.e. links fail, etc.), these topology changes must be simulated by a tester. The second 
class of test behavior is to introduce simple errors into all packets of the same type 
(e.g. modify the network mask in all hello packets). Such behavior can be used to test 
how OSPF deals with packets, that contain invalid values in some fields. The third class 
of behaviors is needed to model more complex tests. Some tests require the tester to be 
aware of the state of an adjacency and to modify only cetain packets or originate new 
ones dependent of this state (e.g. modify the sequence number of the fifth database 
description packet). For some test cases it may also be necessary to combine these 
behaviors. So, all the test behaviors are implemented in appropriate protocol sessions, 
which can be combined in a protocol graph. All the protocol sessions for the testers 
are implemented in the <tt>SSF.OS.OSPFv2.test</tt> package.<p>

<h2>Structure of the Test Suite</h2>

The OSPF functionality can be divided into several areas, which are mostly independent of 
each other. These areas are 
<ul>
<li>the hello protocol,
<li>database exchange and flooding, 
<li>origination of different LSA types and 
<li>routing calculations. 
</ul>
Additionally, {\em general configuration and packet formatting} issues can be seen as a 
separate group. The IOL test suite is divided in exactly these groups. So the test suite 
can be seen as a collection of five subsidiary test suites, which can be executed mostly 
separate from the others. Each of these five test suites again consists of several test
cases. <p>

<h2>Structure of the Test Cases</h2>

Each of the test cases consists of a <i>test setup</i> and a <i>test procedure</i>. The 
test setup describes the network topology for this test. The test procedure describes 
in detail, how the routers should behave and which results should be observed. A basic 
DML configuration can be directly derived from the test setup. A detailed configuration 
and required behavior of the components can be obtained from the test procedure. During 
the simulation of a test scenario, OSPF packets must be logged, where necessary. For the 
evaluation of the test results, these logs can then be analyzed and a verdict can be 
assigned. In summary, the execution of each test case is divided into two phases:

<ol>
<li>simulation of the scenario
<li>evaluation the logfiles and assignment of a verdict
</ol>

The evaluation of results is done automatically by a PERL script which is provided with 
every test case. These scripts extract the necessary information from the logfiles of a 
test scenario and generate a verdict out of this information.

<h3>Assigning Verdicts</h3>

There are three possible verdicts, which can be assigned to every test case. <i>PASS</i> 
means, that the implementation performed exactly as expected. <i>FAIL</i> is assigned, 
when there were offences against the specification. The <i>INCONCLUSIVE</i> verdict is 
assigned, if it is not clear if the implementation complies to the specification. This 
can be the case when log files are missing or some prerequisites for the test are not 
fulfilled (e.g. when testing sequence numbering in the database description process, 
but no database description packets are sent at all). The evaluation of logfiles strongly 
depends on the test cases, so every test case has its own evaluation script. For each 
of the test cases a file named \texttt{report.log} is produced, which contains information 
about which test steps passed or failed. This file also contains the final verdict for 
the test case. To be able to evaluate the reports automatically, the final verdict is 
contained in a line which is formatted as follows:

<pre>
### VERDICT for &lt;test case name&gt;: &lt;PASS|FAIL|INCONCLUSIVE&gt; ###
</pre>

When all test cases of a test suite have been executed, all the verdicts must be examined to
assign a final verdict for this test suite. When all subsidiary test suites have been 
executed, the verdicts for the particular test suites are examined and a final conformance 
statement is produced. Another PERL script (\texttt{verdict.pl}) is used for the assignment 
of verdicts for a test suite. <p>

<tt>verdict.pl</tt> takes the names of the test cases as parameters. The reports of all 
test cases specified in the command line are scanned. If this report exists, it is 
scanned for the above line, which contains the verdict for the test case. If the file 
does not exist or if it doesn't contain a line as specified above, the test case result 
is considered <i>INCONCLUSIVE</i>. The syntax of verdict.pl is as follows:

<pre>
verdict.pl &lt;test_suite_name&gt; &lt;report_file&gt; &lt;test1&gt; ... &lt;testn&gt;
</pre>

The final verdict for the test suite is generated as follows: If all test cases have passed, 
the whole test suite gets a <i>PASS</i> verdict. If one or more <i>FAIL</i> verdicts have been 
found, the whole test suite gets a <i>FAIL</i> verdict. If there are one or more 
<i>INCONCLUSIVE</i> verdicts, but no <i>FAIL</i> verdict, the whole test suite's verdict 
will be <i>INCONCUSIVE</i>. <tt>verdict.pl</tt> also generates an output, which contains a
verdict line as specified above. This output can again be redirected into a file called 
<tt>report.log</tt>. So <tt>verdict.pl</tt> can also be used to produce the final conformance 
statement.<p>

<h2>The Testsuite Dictionary</h2>

Many of the test scenarios require the same or similar configurations for some components. 
These common configurations of components are stored in the testsuite dictionary 
<tt>testsuite.dml</tt> to keep the test case DML configurations simple. The dictionary 
consists of several parts:<p>

<ul>
<li>The <i>interfaces</i> part defines some common interface types (e.g. 100MBit or 1GBit).
<li>The <i>protocols</i> part defines common configurations for <tt>ProtocolSessions</tt>.
<li>Common router configurations are contained in the <i>routers</i> section.
<li>The <i>\em networks</i> part finally contains several networks of different sizes.
</ul>

In the routers section there are common router configurations with different numbers of 
interfaces running different protocol configurations. Some test scenarios of the IOL test 
suite require the router to send a minimum number of LSAs. There are two approaches for 
realizing such scenarios. A tester could generate "virtual" LSAs to fill its link state 
database. In order to leave the OSPF session unchanged, this would require to simulate the 
complete building of an adjacency with a router connected to a virtual network. That means, 
that the tester has to implement parts of the OSPF protocol itself. The second approach to 
this problem uses the fact, that large networks can be generated easily in SSFNet. So, the 
router which has its link state database to be filled is simply connected to a network, in 
which a sufficient number of LSAs are generated (see Figure 3). In the test dictionary, 
there are networks with 8, 57 and 400 routers, which can be used for this purpose.<p>

<div align="center"><img src="pictures/filling_lsdb.jpg" width=225 height=201><br>&nbsp;<br>
<b>Figure 3</b> - Filling a router's link state database</div>

<br><hr>
<div align="center"> 
<a href="http://ssfnet.d-jacob.net">Home</a> | 
<a href="index.html">Up</a> | 
<a href="selection.html">Previous</a> | 
<a href="ospfv2test.html">Next</a>
</div>
<hr><br>

<i>Last update: 2002-02-05, Dirk Jacob (<a href="mailto:dirk@d-jacob.net">dirk@d-jacob.net</a>)</i>
</body>
</html>